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ABSTRACT 



Pursuant to the Council of Ontario Universities' procedures 
for periodic quality reviews of undergraduate programs, auditors examined the 
policies and programs of the University of Ottawa (UO) during 1997. The 
English, Chemistry, and Physiotherapy departments were selected for audits of 
their program review procedures; the Environmental Studies program was 
selected as a sample of a new program. After examining UO's review policies 
and program review records, and finding that mechanisms were in place to 
implement recommendations, the auditors held a site visit. Following a brief 
introduction and description of the methodology used in the audit, the report 
includes the following sections: (1) a description of UO program reviews,* (2) 

the general guidelines for program reviews at UO; (3) conformity of the UO 
review process for existing programs with Audit Committee procedures; (4) 
conformity of UO policy for implementing new programs with the Audit 
Committee framework for new programs,* (5) the report’s conclusions, 
indicating that UO's review program predated the Audit Committee framework 
and was in a state of flux at the time of the audit; and (6) a summary of 
recommendations and suggestions. (Contains the schedule for the first cycle 
of audits (1997-2003), lists of auditors and UO principal documents, and site 
visit schedule.) (MAB) 
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Glossary of Acronyms and Terms 



COU Council of Ontario Universities 

MET Ministry of Education and Training 

OCAV Ontario Council of Academic Vice-Presidents 

OCGS Ontario Council on Graduate Studies 

SCEUP Senate Committee on the Evaluation of Undergraduate Programs, 

University of Ottawa 

SIRP Service for Institutional Research and Planning, University of Ottawa 

UPRAC Undergraduate Program Review Audit Committee 



UPRAC Procedures Undergraduate Program Review Audit Committee Procedures refers to 

the OCAV Constitution, Bylaw 1, Appendix A. 

UPR Framework . . . Undergraduate Program Review Framework refers to the Objective, 

Structure and Elements of the Undergraduate Program Review Procedure 
found in the OCAV Constitution, Bylaw 1, Appendix A, Section 1. 
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1 Introduction 

In October 1996, the Council of Ontario Universities (COU) approved the establishment of a 
procedure for the systematic auditing of the policies and processes in place at all Ontario universities 
for the conduct of periodic quality reviews of undergraduate programs. The procedure and guidelines 
specify that auditing of processes includes the examination of a representative sample of the quality 
reviews. Subsequently, in February 1997, the guidelines were amended to include the auditing of the 
mechanisms used by the universities for the implementation of new undergraduate programs. 
Authority for the organization and management of the audits is vested in the Ontario Council of 
Academic Vice-Presidents (OCAV). The detailed oversight of the audit procedure is devolved to a 
committee of OCAV, the Undergraduate Program Audit Review Committee (UPRAC) whose 
responsibilities are set out in Bylaw 1 of the OCAV Constitution. One of UPRAC’s duties is to 
recommend to OCAV the schedule of audits, which started in 1997 with Brock University and the 
University of Ottawa. The schedule for the first cycle was approved by OCAV at its meeting of 
February 27, 1997 and is displayed in Appendix 1. 

The audits themselves are conducted at arm’s length by at least three auditors who are appointed by 
UPRAC according to the criteria in the bylaw: “Auditors shall be chosen for their recognized strength 
in the development and operation of undergraduate programs. They shall not hold an administrative 
appointment in an Ontario university during their terms as auditors.” (See Appendix 2 for the names 
and affiliations of the Auditors for 1997.) The procedures to be followed by the Auditors are spelled 
out in the Procedures for the Audit of Undergraduate Program Reviews, Appendix A of the bylaw, 
hereafter called the UPRAC Procedures. It describes in some detail “the objective, structure and 
elements” that “any credible periodic undergraduate program review procedure undertaken by an 
institution must include.” For convenience, these key review components can be described as the 
Undergraduate Program Review Framework, and will be referred to hereafter as the UPR 
Framework. 

A critical caveat having to do with the timing of the introduction of the UPRAC Procedures and the 
first few institutional audits must be entered here. Those universities whose review policies and 
practices are being audited in the early years of the first cycle of audits will have had in place review 
policies that predate the approval of the UPRAC Procedures. This means that the program reviews 
sampled for the audit will have been conducted following institutional policies and processes that may 
not incorporate every aspect of the UPR Framework. Within these constraints, the Auditors could 
not apply the UPR Framework retroactively as a standard against which the university’s practices 
must be measured. The components of the UPR Framework have, therefore, been used as markers 
of “best practice” and as signposts for improvements. Specific suggestions and recommendations are 
accordingly made with a formative intent. 
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In framing the report and presenting their findings, the Auditors have found it helpful to distinguish 
between recommendations and suggestions. Instances where the Auditors consider the policies and 
procedures not to be in conformity with the UPR Framework are cast as recommendations. They 
have made suggestions in cases where, although the review process is in conformity with the UPR 
Framework, they think the process could be improved. It will be noted that some recommendations 
and suggestions are, in fact, generic in nature and may be relevant to many institutions. 

The Auditors also wish to record that they have found some parts of the UPR Framework to be open 
to interpretation, and for that reason they have applied it with “interpretive charity.” It should be 
emphasized that in no way do they think that this has weakened the audit process. To the contrary, 
just as there is room for improvement in the way in which institutions conduct their quality reviews, 
so too the UPR Framework may be improved. The Auditors will be reporting separately to OCAV 
on those aspects of the procedure that should be modified or clarified. 



2 Methodology and Verification Steps 

In preparation for their work, the Auditors requested from the University of Ottawa a copy of the 
current policies for conducting undergraduate program reviews and implementing new programs, plus 
the schedule of programs that had been reviewed and those slated for future review. 

To satisfy the injunction in the UPRAC Procedures that “the Auditors will take into account the 
diversity of educational programs offered by the university, whenever possible,” the reviews of three 
Ottawa programs were chosen for the audit, one from the humanities (English), one from the physical 
sciences (Chemistry), and one professional program (Physiotherapy). The program in Environmental 
Studies was selected as an example of a new program. 

As outlined in the UPRAC Procedures, the university was requested to provide “general documents 
that present the [undergraduate program] review practices and policies adopted and implemented by 
the institution” and “any additional documents they deem appropriate.” Relevant documentation 
would include: 

• the internal review policies in place at the time of the review of each program where they 
differed from those currently in force; 

• the complete record of the reviews chosen for audit: 

• the programs’ self-appraisal documents; 

• the evaluations of external experts; 

• the appraisals by peers internal to the university but outside the program under 
review; 

• the responses, if any, to the evaluations at all levels; 

• the minutes of any discussion at Senate (or one of its standing committees); and 
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• evidence that mechanisms are in place to act on the recommendations arising from the 
process. 

(A list of the principal documents provided by the University of Ottawa is included in Appendix 3.) 

Having met and reviewed the documentary record, the Auditors paid a site visit to the university on 
May 13, 1997. During the visit, they met with the incumbent Vice-Rector, Academic and Research; 
the Vice-Rector, Academic-designate; the deans of the three faculties; and the chairs and 
representatives of the departments whose program reviews had been selected for audit. They also 
met the chair and members of the Senate Committee on the Evaluation of Undergraduate Programs. 
(The schedule of meetings for the site visit is provided in Appendix 4.) 

The Auditors would like to record their appreciation for the assistance and cooperation they received 
from all those involved with their visit and the provision of documentation. Everyone concerned 
approached the task with an understanding of its importance; it was viewed not as an intrusion, but 
rather as an opportunity to reflect on what it is that universities can and should be doing to improve 
their undergraduate programs. 



3 Program Reviews at the University of Ottawa 
3.1 Existing Programs 

The University of Ottawa has a long history of considering the structure and development of its 
undergraduate programs on an ad hoc basis. In the early 1990s, it was decided that reviews 
should be conducted on a regular and formal basis, and in 1992 the University Senate approved 
the policy that sets out the current review requirements and process. This document, entitled 
Evaluation of Undergraduate Programs, specifies: “The evaluation of undergraduate programs 
will take place, in principle, every seven years.” The schedule for the reviews, completed and 
planned is contained in the document “Evaluation des programmes de ler cycle,” dated 20 
novembre 1996. This latter document quotes from the policy an important feature of 
undergraduate reviews at the university: “To make the best use of the time, energy and resources 
already required by other periodical evaluation processes that the University must conduct, the 
undergraduate evaluation will take place during the same year as the periodic evaluation of 
graduate study programs conducted by the Ontario Council on Graduate Studies (OCGS) and/or 
the periodic evaluation leading to the professional accreditation of certain study programs, 
provided the unit must undergo these evaluations also.” 

The first cycle of reviews under this policy began in 1993-94 and 17 reviews were completed in 
the first three years. In 1996-97, six reviews were in process, one of which was carried over 
from 1995-96. Interdisciplinary programs (which are not freestanding) at Ottawa are scheduled 
along with the other programs. The schedule calls for the completion of the first cycle of reviews 
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by the end of the academic year 1999-2000 (the year still to be set for three programs). The 
cycle will then be repeated. 

Schedule of Reviews of Existing Undergraduate Programs at the University of Ottawa 



■Faculty 


Com pletedif • 
1993-1996 


In Progress X 
1996-1997 


• Planned J 
•Xlvlll 1 997-2000 ; 


Administration 


— 


— 


2 


Arts 


5 


3 


12 


Education 


— 


— 


1 


Engineering 


2 


1 


2 


Law 


2 


— 


— 


Medicine 


— 


1 


— 


Science 


3 


— 


2 


Health Science 


2 


1 


1 


Social Science 


3 


— 


3 


Interdisciplinary 


— 


— 


2 


Total 


17 


6 


25 



3.2 New Programs 

According to the guidelines set by the Ministry of Education and Training (MET), the 
implementation of new programs in Ontario universities should be preceded by their introduction 
in universities’ five-year plans. Once introduced in the five-year plan, new programs are 
implemented by universities as resources permit and opportunities arise. Therefore, the time 
frame during which new programs pass from the stage of introduction in the university’s five- 
year plan to implementation varies. At the University of Ottawa, one new program was 
introduced in 1995-96 and three in 1996-97. 
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4 General Principles and Guidelines Used for Undergraduate Reviews by the 
University of Ottawa 

4.1 Existing Programs 

Ottawa’s policy governing the review of its undergraduate programs is contained in the 
document Evaluation of Undergraduate Programs, which was adopted by Senate on November 
2, 1992. It is organized in three sections that are described below. 

4.1.1 Objective 

The overarching objective of the review process is stated as “maintaining and, where needed, 
improving the quality and relevance of . . . bachelor and certificate programs.” As noted in the 
policy, evaluations are to deal with: 

(1) how objectives that are specific to programs at this level are attained; 

(2) how well these programs suit not only the needs of various student clienteles who 
register for them, but also the specific objectives of the University of Ottawa in the area of 
bilingualism and services to the Francophone population of Ontario; 

(3) how well the quality and quantity of available human and material resources meet the 
needs of these programs. 

4.1.2 Process 

The conduct of periodic undergraduate reviews at the University of Ottawa “takes place under 
the authority of the Vice-Rector, Academic,” who is assisted by the Senate Committee on the 
Evaluation of Undergraduate Programs, which “advises the Vice-Rector on the evaluation 
schedule, on the choice of external examiners, on the conclusions to be drawn from the 
evaluations and on any other question dealing with the process overall.” The committee itself 
is broadly representative of programs within the university, its maximum of five members being 
drawn “from the two major study areas on campus (pure and applied natural sciences [sic] and 
humanities and social sciences).” 

The policy specifies the following seven steps in the undergraduate review process: 

(1) Timing and coordination of internal and external reviews. Reviews of undergraduate 
programs take place, in principle, every seven years. As noted above, the University of Ottawa 
seriously attempts to coordinate its internal process with OCGS graduate appraisals and 
professional accreditations, where these apply. However, the policy recognizes that these other 
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evaluations may have different criteria, and it states that undergraduate reviews must be 
conducted according to the specific objectives set for them. 

(2) Service for Institutional Research and Planning (SIRP) data and analysis. To prepare for 
the review, and to assist those participating in the reviews (members of the program in their self- 
appraisal, external evaluators and the Senate Committee) to realize the objectives of the process, 
the University of Ottawa’s Service for Institutional Research and Planning provides a number 
of basic data on students, grade distributions, teaching evaluations and the results of a survey of 
recent graduates. At the request of the dean, further data can be provided. 

(3) Self-evaluation by participants in the program under review. Self-evaluation is to involve 
“the entire teaching staff of the unit as well as the students on the council and on the standing 
committees of the council.” The evaluation is to cover the same subjects as those submitted to 
the external evaluators. The point of the exercise is to “allow the parties involved to reassess 
and, where needed, rethink the programs under study.” Finally, the product of the exercise is 
“a document that will provide background information for the external evaluators.” 

(4) Selection of external evaluators. The selection of evaluators, normally from outside the 
university, is made by the Vice-Rector, Academic, based upon a list established by the Senate 
Committee from suggestions made by the dean of the faculty, the unit responsible for the 
programs concerned and the Undergraduate Studies Council. The evaluators must be from the 
same discipline as that of the academic unit concerned. 

(5) Site visit. The site visit “usually stretches over two days” and the evaluators are to visit at 
the same time, wherever possible. They are expected to meet with the Vice-Rector, Academic, 
the dean and the chair of the unit concerned, the full-time professors and representatives of the 
part-time academic staff, as well as a representative sample of students and key members of the 
support staff. 

(6) Evaluators ’ reports and feed back. Following the site visit, the evaluators submit individual 
reports to the Vice-Rector, Academic, who relays copies to the dean and the chair of the unit 
involved. They, together with the unit itself, are invited to submit additional comments. 

(7) Conclusions drawn from all the evidence. The Senate Committee on the Evaluation of 
Undergraduate Programs drafts for the Vice-Rector, Academic, “notice on the conclusions that 
can be drawn from [each] evaluation,” based on the total evidence detailed above. 

4. 1 .3 Specific subjects to be evaluated 

The Ottawa policy specifies the subjects to be assessed by the units concerned and the external 
evaluators as follows: 
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(1) The specific objectives of undergraduate study programs; 

(2) ... how undergraduate study programs meet the needs of the student clienteles and how 
they comply with the specific objectives of the University of Ottawa; and 

(3) ... the quality and quantity of the means made available to the undergraduate study 
programs (concentration, honours, certificate) offered by the academic unit. 

In addition to elaborating on each of these subjects, the policy contains for further guidance an 
appendix in which they are all developed in detail. 

4.2 New Programs 

The university’s policy and practice for the introduction of new programs is now guided to a 
large extent by the current requirements of MET as set out initially in the memorandum dated 
November 8, 1996 from the Assistant Deputy Minister to the Executive Heads and reconfirmed 
in a subsequent memorandum on June 24, 1997 from the Director of the Universities Affairs 
Branch. The programs approval process is described as one of self-administered regulation and 
calls for the institutions to certify that their proposals meet six criteria: program nomenclature 
(or “truth-in-advertising”), academic quality, financial viability, institutional appropriateness, 
student demand and societal need. A seventh criterion, duplication, is evaluated by the Ministry. 
(“Core” undergraduate Arts and Science programs are exempted from this process.) 

Internally, the Council on Undergraduate Studies considers proposals from departments and 
faculties for new programs. It recommends the approval of these programs to several bodies: 
the Academic Planning Council, the Executive Committee of Senate, and Senate. 



5 Conformity of the University of Ottawa Review Process for Existing 
Undergraduate Programs to the Expectations of the UPRAC Procedures 

The UPRAC Procedures specify that the report of the Auditors will include comments on how well 
the procedures and process of the institution under review conform to the objectives, structure, and 
elements they establish (the UPR Framework). They also specify that the Auditors should provide 
details on “how well the review procedure and process conform to the institution’s own policies.” 
This provision is especially important at the inception of the audit process. 

As all the program reviews chosen for audit in 1997 were conducted prior to the adoption of the 
UPRAC Procedures, a negative judgment relative to that standard would be both uninformative and 
unfair, since the local practices could not have been designed with the UPR Framework in mind. 
Second, it was evident to the Auditors that local review procedures are in a state of flux, and that 
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actual practice has in many regards outstripped the policy statement. Finally, while the expectation 
is that all Ontario institutions will generate program review policies that reflect at least the minimum 
requirements of the UPR Framework (if they have not already done so), it is now, and will doubtless 
always be, the case that, to accommodate universities’ diverse needs, local review policies will go 
beyond the minimum that the UPR Framework requires in different ways that call for verification. 

These comments on conformity are organized into two parts: 

• conformity of the University of Ottawa’s procedures for program reviews to its policy, and 

• conformity of the University of Ottawa’s policy for program reviews to the UPR Framework. 

5.1 Conformity of the University of Ottawa’s Procedures for Program Reviews to its 
Policy 

Here the comments on conformity are organized according to the structure of the Ottawa policy 
document. 

5.1.1 Objective 

On the basis of their examination of the conduct of reviews in the three programs, the Auditors 
can assert with confidence that the Ottawa evaluation of undergraduate programs is being 
conducted consistently with its paramount objective. From the reviews they selected, it is 
evident to the Auditors that the reviews were conducted, for the most part, with the purpose not 
to defend the status quo, but to determine what was working and what was not from the vantage 
of the programs’ distinctive character and the students’ particular needs. The Auditors noted 
that the people from programs that had undergone review were refreshingly open to constructive 
feedback from external experts. It was clear in each review that the feedback was treated very 
seriously, that the program reviews were of good quality, and that whatever defects were 
uncovered had been rectified or were in the process of being addressed. More than one 
respondent during the site visit remarked that the effectiveness of the review process was 
critically dependent on the commitment of the senior administration of the university, and that 
such commitment was forthcoming and perceived to be there, a view that was confirmed by the 
Auditors. 

5.1.2 Process 

(1) Timing and coordination of internal and external reviews. 

This provision had application to all three reviews. The Chemistry and Physiotherapy programs 
are both subject to external accreditation procedures, and the English and Chemistry programs 
are attached to graduate programs subject to OCGS appraisals. In any case, it was not always 
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possible for the university to schedule the internal and external reviews “during the same year,” 
as the policy foresees. 

There are two timing and coordination provisions in the Ottawa policy, and though they are 
supposedly related as ends to means, they do not always have the same implications. The policy 
seeks “to make the best use of time, energy, and resources already required by other periodical 
evaluation processes that the University must conduct,” and mandates the conduct of the internal 
reviews in the same year as the external one. 

In conversations with representatives of units that had been reviewed, it became clear that 
requiring units to undergo internal and external reviews in the same year, while admirable in 
theory, has two principal drawbacks in practice. First, for the small unit, the task of meeting all 
the requirements of reviews that are sequenced closely together was highly stressful. Further, 
they found it difficult to do justice to both reviews, in addition to their other teaching, scholarly 
and administrative duties. 

Second, while a number of the information requirements for the internal review process and 
external OCGS and accreditation exercises are the same or similar, and there is an economy 
preparing the necessary documentation for both simultaneously, in other respects the processes 
are quite unlike, and it is necessary for those undergoing review to keep the differences clearly 
in mind. Accreditation exercises typically verify that a program satisfies minimum necessary 
professional standards. OCGS appraisals, although designed to lead to the improvement of 
graduate education, are sanctioned by similarly summative judgments. By contrast, the Ottawa 
undergraduate review policy focuses intensively on the needs of undergraduate students, the 
extent to which the programs under review are meeting those needs, and how effectively the 
human and material resources of the university are being allocated to support these programs. 
It is, of course, not impossible that units undergoing multiple reviews simultaneously keep in 
mind their different purposes. However, it does make it more difficult, and for that reason it is 
important to reinforce the different features of the reviews. 

In summary, the departmental representatives, in one case quite forcefully, were of the view that, 
if the purpose is to make best use of the available time and resources, then requiring all reviews 
to be held in the same year was not the best way to do so. 

Suggestion 1: It is suggested that the Senate Committee on the 
Evaluation of Undergraduate Programs (SCEUP) and the Vice-Rector, 

Academic, together review the appropriateness of the requirement that 
internal reviews and OCGS appraisals and external accreditation 
exercises “take place during the same year,” if the purpose is to “make 
best use of the time, energy, and resources already required” for the two 
kinds of external process. 
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Though there is no reason to think that any particular prescription will necessarily fit all cases, 
those who preferred some alternative to the present arrangement, when pressed, invariably 
suggested that the internal review take place in the year or two preceding the external process. 
This arrangement would allow the gathering in advance of the necessary documentation, and 
would allow relatively risk-free correction of shortcomings that might be uncovered before any 
kind of external assessment is conducted. 

(2) Service for Institutional Research and Planning (S1RP) data and analysis. 

Though the Auditors did not see a complete set of data from any of the reviews, based on the 
self-analysis, external reports and responses of the Senate Committee, it is clear that this is a key 
provision of the Ottawa policy and that the information was useful in the three selected reviews. 
Because it is so important in allowing all those concerned in the process to reach conclusions 
based upon empirical evidence, a suggestion is made that emerged out of remarks voiced by 
some of the units whose reviews were audited. 

Suggestion 2: Since all academic units are subject to internal reviews 
and since the provision and the analysis of data is an important 
component of the quality evaluation, it is suggested that SIRP develop 
a method for providing the required data in a simplified and at least 
partially analyzed format, based on common algorithms for all units. 

SIRP might also make workshops available on reading and 
understanding the material it produces, and the kinds of questions that 
can be asked of the data. 

(3) Self-evaluation by participants in the program under review. 

In general, this part of the process was done well, and in conformity with the four requirements 
of the Ottawa policy [see 4. 1 .2 (c)]. In the generation of the report, there was least consistency 
in the level of participation in general and of student inclusion in particular, but differences seem 
ascribable both to variations in scale and in local culture. Units undergoing review should be 
encouraged to approach the process as an opportunity to engage in critical reflection about their 
current state and how improvements might be made, and not as a means to feature their 
accomplishments. The recent practice of providing workshops by the Office of the Vice-Rector 
should assist units coming up for review in achieving a high level of preparedness that helps to 
ensure that the external feedback is useful and relevant. 

(4) Selection of external evaluators. 

The Ottawa procedure specifies that the two evaluators are “normally” from outside the 
university. All of the evaluators in the three instances examined conformed to this rule, and the 
Auditors were assured that it would be a most unusual circumstance that would ever lead to such 
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an exception. The department representatives were not always clear on the process in place to 
make the ultimate choices, which seemed occasionally to depart from the suggestions made by 
the units themselves, but no objections were raised about the identity of those ultimately chosen. 

Suggestion 3: Since the practice at Ottawa has been to retain external 
evaluators in all cases, it is suggested that the word “normally” be 
removed from the policy document. This would strengthen the policy 
and remove any possible doubt as to the intentions of the university. 

(5) Site visit. 

The Ottawa policy specifies that the site visit “usually stretches over two days” and specifies 
those who must be included in the itinerary. Interestingly in the audit sample, two out of three 
programs had site visits lasting only one day. Though in no case was the exercise rendered 
nugatory by the shortness of time, it was clear that both the reviewers and those under review 
would have profited from the additional time afforded by the second day “usually” provided. 

(6) Evaluators ’ reports and feedback. 

There seem to have been slight misunderstandings about evaluators being expected to provide 
individual versus joint reports, and deans’ comments being made available to inform the Senate 
Committee deliberations, but these minor matters were all corrected. 

(7) Conclusions drawn from all the evidence. 

This step was completed in every case that was audited, and, indeed, the Ottawa practice seems 
to go beyond what its policy requires. The current policy suggests that the role of the Senate 
Committee on the Evaluation of Undergraduate Programs at the end of the process is to draft 
for the Vice-Rector, Academic, “notice on the conclusions that can be drawn from the evaluation 
of the bachelor programs in question” based on kinds of evidence that are explicitly specified. 
In more than one instance, however, the emphasis of the committee’s notice was very different 
from the remarks made in the evaluators’ reports and the comments they evoked, and, in rare 
instances, there seemed to be no basis in the documentation for the conclusions reached by the 
committee. There was nothing necessarily improper about such conclusions, except that they 
reflect the way in which the Senate Committee has evolved from being the beginning of the suivi 
to being a representative committee of peers within the university but outside the program under 
review, which provides an additional independent assessment (and not just a judgment of the 
conclusions to be drawn from all the other evidence). 

Though it is not the business of the Auditors to review the programs, as opposed to the review 
process, it should be recorded that, based on the evidence at their disposal, they have no doubt 
that the three programs whose reviews were audited are very good programs. However, a casual 
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reader of the minutes of the Senate Committee, or of the communications drawn from those 
minutes by the Vice-Rector, Academic, would be hard pressed to discover how good in fact they 
are. It is important both for units within the university, and for those interested in such matters 
outside the university, to be reassured where this is the case. A good place to start is in the notice 
of the conclusions that are drawn by the Senate Committee. Even where there are deficiencies 
that require corrective measures, some context of their significance would be helpful to the 
“programs in question.” 

The role and functions of the Senate Committee are discussed again in sections 5.2.2 (3) c) and 
5.2.2 (6) where recommendations are made. 

5.1.3 Specific Subjects to be Evaluated 

The layout of the subjects to be addressed both by programs and by evaluators seems to 
distinguish between three general areas contained in the body of the policy, and a larger set of 
more specific questions included in the appendix. This organization proved useful to the 
programs chosen for review, all of which in various ways treated the policy and the appendix as 
either a template or an organizing principle for their own self-appraisal reports. 

The University of Ottawa’s “Specific subjects to be evaluated” and the illustrative questions 
appended are in many respects models of their kind. In practice, however, in the three reviews 
that were sampled, the perspective on the part of all concerned was oriented to the vantage of 
the discipline involved. This is both understandable and generally commendable; however, 
notwithstanding the relevant questions in the appendix of the Ottawa policy, it is, in one respect, 
out of phase with the student perspective that the Ottawa policy seems (rightly) to emphasize. 
To the extent that the units under review and evaluators consider the courses taken by students 
in other programs, they approach issues from the vantage of the producers, as it were, rather than 
consumers. Having “out-of-program” courses considered by evaluators as part of the reviews 
of the programs whose students are taking the courses would be more consistent with Ottawa’s 
commendable student-learning perspective. 

Suggestion 4: In a context where all programs within the university are 
to be assessed within a seven-year cycle, it would make sense for 
programs under review and external evaluators to be enjoined to 
consider critically whether or not the needs of students within the 
program being reviewed are being met by the courses either required or 
recommended in other departments of the university. 

The Auditors conclude, based on examination of the documentation provided, and interviews 
with interested parties, that the procedures and process in place at the time the reviews were 
conducted were observed in the three cases that were audited. 
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5.2 Conformity of the University of Ottawa’s Policy for Program Reviews to the UPR 
Framework 

In making recommendations and suggestions and for improving what are already a very sound 
policy and procedure, the Auditors once more wish to emphasize their awareness that the 1992 
policy — and the program reviews that were audited — predate the UPRAC Procedures. The 
conformity review which follows evaluates the Ottawa policy against the objective, structure and 
elements of the Undergraduate Program Review Procedure contained in the UPRAC Procedures. 
These components of the UPR Framework appear in italics below. 

5.2.1 Objective of the Undergraduate Program Review Procedure 

The objective of the institutional periodic undergraduate program review procedure is to 
assess the quality of the undergraduate programs that the university provides in all areas 
of study. 

The Ottawa policy begins with a clearly stated objective that conforms well to this general 
expression of the objective of undergraduate program reviews. It was obvious from all of the 
meetings with the Auditors that those involved understood the overall objective and its three 
more specific parts. 

5.2.2 Structure of the Undergraduate Program Review Procedure 
The institutional undergraduate program review policy statement should: 

( 1 ) identify an authority responsible for application of the policy ; 

The Ottawa policy states: “The evaluation of undergraduate study programs takes place under 
the authority of the Vice-Rector, Academic, who is supported in this task by the Senate 
Committee on the Evaluation of Undergraduate Programs.” In practice it is exercised well by 
them. 

(2) define the undergraduate program review criteria; 

As indicated above in section 4, the University of Ottawa policy and appendix are particularly 
explicit on the criteria for evaluation. 

(3) prescribe at least the following steps: 

a) self -appraisal by professors and students participating in the program; 
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Not only does the Ottawa policy explicitly call for this measure, but it goes into some detail on 
what this involves (who is to participate in the self-appraisal; its point or purpose; the subjects 
to be dealt with; and the kind of document that should result). Also, the institution now provides 
workshop training in self-appraisal for units about to undergo the review process. 

b) evaluation by at least one expert from another university, including universities 
outside Ontario; 

The Ottawa policy calls for not one, but two persons from the same discipline as the unit under 
review, who are normally (but in practice have always been) from outside the University of 
Ottawa. 

Recommendation 1: Though there is no evidence to question the 
impartiality of the external evaluators who have been chosen by the 
University of Ottawa, the auditors recommend that all Ontario 
universities that have not already done so enunciate as part of their 
program review policy the principle that external evaluators will be at 
arm’s length from the program under review, along with their 
interpretation of what is entailed by the principle (not relatives, 
collaborators, supervisors/supervisees, and so on). 

c) appraisal by peers chosen among professors from the institution who do not 
participate in the program under review. 

This requirement in the UPR Framework calls for an interdisciplinary perspective to ensure that 
university-wide standards for review are met in all cases. How this requirement is met by 
universities may vary according to size, structure and ethos. Though the Ottawa policy is not 
entirely clear on this matter, it is understood by the Auditors, based on the site visit, that this role 
is to be filled for all reviews by the Senate Committee, whose membership represents a cross 
section of programs offered by the university. However, it is the view of the Auditors that the 
committee’s role and functions need to be more clearly defined if it is to have the truly evaluative 
purpose envisaged in the UPR Framework as “appraisal by peers chosen among professors from 
the institutions who do not participate in the program under review.” The university may, of 
course, choose to have this particular function assigned to another body or bodies, if it is thought 
to be too demanding given the number of programs that have to be reviewed annually. 

Recommendation 2: If the Senate Committee on the Evaluation of 
Undergraduate Programs is to play the role of an appraisal committee of 
peers from within the university, its terms of reference must be explicit 
about that role and function. In fulfilling that role, the Senate Committee 
must review the self-study brief, the external evaluators’ reports as well 
as the department’s response in order to arrive at a conclusion on the 
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quality of each program and determine the actions to be taken as a result 
of the review. 

d) the university may also, where appropriate, seek the advice of others such as 
representatives of industry, professions and practical training programs; 

The auditors raised this matter during their site visit. From one informant in particular, they 
heard that access to advice from representatives of industry would have been particularly helpful 
with respect to the co-op program that had been recently initiated in his unit. As this step is 
optional only, it is for the University of Ottawa to determine for itself whether or not such a 
measure would be useful. 

Suggestion 5: In reviewing its policy, the university should consider 
making available to programs undergoing review the option of seeking 
advice from representatives of industry, professions and practical training 
programs when it is regarded as useful by the unit under review. If this 
option is introduced, the policy should make it clear that there must 
always be at least one evaluator who is an academic even when the 
advice of others is sought. 

(4) specify the procedures for action on the recommendations arising from the 
undergraduate program review; 

The final step in the University of Ottawa policy section on process is the requirement that the 
Senate Committee draft for the Vice-Rector, Academic, “notice on conclusions that can be 
drawn from the evaluation of the bachelor programs in question.” In practice, however, this has 
not been the end of the process. The Vice-Rector, Academic, has communicated the results of 
the Senate Committee’s deliberations, in the form of a letter informing the unit of the follow-up 
action it was expected to take, with provision for a report within two years on the progress being 
made on corrective measures. 

Because of the timing of the site visit, it was not possible to corroborate that the progress reports 
with a deadline of June 1997 would be received in good order and would be adjudged 
satisfactory at the required time. Indeed, it came as a surprise to the representatives of some units 
that they were expected to report with a June 1997 deadline. No one, however, thought that 
reporting satisfactory progress based on the follow-up action taken since the review exercise 
would constitute any kind of problem. Here is another example of where the actual practice of 
evaluation has evolved beyond the policy statement adopted in November 1992. 

Recommendation 3: The Ottawa evaluation policy should be revised 
to make explicit provision for “procedures for action on the 
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recommendations arising from the undergraduate program review,” as 
required by the UPR Framework, and reflecting actual practice. 

Additionally and as a helpful reminder, the secretariat supporting the Senate Committee might 
send timely reminders to those units with any requirements placed on them to document follow- 
up action. 

(5) determine the periodicity of the review cycle ; this should not exceed 10 years for 
a university ’s full set of programs; 

The Ottawa policy conforms in that it specifies: “The evaluation of undergraduate programs will 
take place, in principle, every seven years.” The Auditors are pleased to note that the review 
schedule for the first cycle (set out in the table in section 3. 1) is being maintained. (One review 
from 1996 took longer to complete than was expected, but it is in the final stages of the process.) 

(6) establish a policy to ensure publication of a summary of the institutional review 
report (excluding all personal information); the summary should include a statement 
of the strengths and weaknesses of the program, and an outline of the main 
recommendations of the review committee. This summary is to be presented to the 
Senate and Board of Governors of the university. 

Though not required by the policy, the Senate Committee does prepare an annual report — the 
Auditors have the report for the 1995-96 period — which summarizes the evaluations that have 
been completed, and provides details on improvements that are required. The Auditors were 
surprised by the terseness of this report, which is presented to the Senate. While it is important 
to identify and correct shortcomings, these must be set in context and it is equally important to 
report on quality and success. 

Recommendation 4: The university’s policy should be revised to make 
clear that a summary report on the results of the internal review process 
will be presented annually to the Senate and the Board of Governors. In 
addition, the Senate Committee should make every effort to ensure that 
its reports deal with both the strengths and weaknesses of the programs 
that have been reviewed. 

5.2.3 Elements of the Undergraduate Program Review 

The institutional procedure for review of existing undergraduate programs should address 
the following points: 

(1) consistency of the program with the general objectives of the institution ’ s mission 
and development plan; 
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This is set out particularly well in the policy document and, as a result, is generally executed 
thoroughly by the programs under review and the external evaluators. 

(2) appropriateness of the admission requirements for the program ’s educational 
objectives ; 

The Ottawa policy is virtually silent on this matter. Where programs raised the issue of student 
preparedness to undertake the program in question, the evaluators pursued the matter, though 
it was difficult to answer the questions because of the nature of the data and analysis provided 
by the SIRP. 

Recommendation 5: The university review policy should include a 
reference to the evaluation of the appropriateness of admission 
requirements where academically warranted. 

(3) appropriateness of the program ’s structure and curriculum for its educational 

objectives; 

There is very full coverage of these kinds of questions in the policy statement. 

(4) appropriateness of mode of delivery and of methods used for the evaluation of 
student progress; 

Specific questions centred upon each of these headings are suggested in the policy. 

(5) appropriate utilization of the existing human/physical/financial resources. 

There is a specific section of the policy and recommended questions regarding these matters, 
which were dealt with in every review that was audited. 

(6) definition of indicators that provide evidence of : 

• quality of faculty 

Ottawa’s policy and practice are noteworthy, since they focus not just on the quality of the 
faculty’s qualifications but particularly on their participation in the program under review. 



# 



student clientele (applications and registrations) 
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SIRP provides units with basic data for students in their programs. The policy also requires units 
to identify the university’s distinctive student clientele, and then to assess whether or not that 
clientele’s particular needs are being served by the programs. 

* student quality 

There does not seem to have been any explicit attempt to deal with this difficult issue. 

• outcomes of the program (graduation rate, length of studies, etc.) 

The two examples of outcomes given above are among a number of issues for which the SIRP 
provides its data and analysis to units, external evaluators, and the Senate Committee to inform 
the self-appraisal, reports and comments on the reports. 

In summary, the Auditors are satisfied that in the area of quality indicators the current Ottawa 
policy complies sufficiently with the UPR Framework’s requirement. When the policy is next 
reviewed the indicators might be reviewed to see if they can be more explicitly defined. The issue 
of quality indicators is a continuing matter of concern. Criteria that measure the quality of 
incoming or departing students or some assessment of the “improvement factor” are issues that 
deserve consideration by OCAV. 

The overall conclusion of the Auditors is that the Ottawa policy and its application conform very 
closely to what is set out in the UPR Framework. As has been pointed out, there are places where 
the practice goes beyond what is called for in the policy and it is mostly in these areas that they have 
made recommendations. 



6 Conformity of the University of Ottawa Policy and Procedures for the 
Implementation of New Undergraduate Programs to the UPR Framework for 
New Programs 

The UPR Framework for the implementation of new undergraduate programs states that: 

a) the institution should provide to the auditors its policy regarding the implementation of 
new undergraduate programs; 

b) the institutional procedures for the implementation of new undergraduate programs 
should address points [a) - e)J above and provide evidence of the quality of the faculty. 

Much of what was said in the opening paragraph of section 5 holds here too. The record of the 
approval of the new Program in Environmental Studies — the one selected for the audit — predates 
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the establishment of the UPR Framework. The Auditors were presented with considerable 
documentation for the introduction of this program. They have no reason to question the 
thoroughness with which it was scrutinized, and the process appeared to follow the procedures 
Ottawa established for the introduction of new programs. However, more detailed guidelines are 
needed in order to allow for an audit of compliance. 

One way of viewing what OCAV has demanded in the UPR Framework is that, consistent with the 
information requirements now imposed by the Ministry of Education and Training for funding 
approval of new “non-core” programs, universities should be taking into account the consistency of 
what is proposed with their more general development plans. Specifically, they should address the 
issue of whether this is an appropriate use of available resources, as well as provide some critical 
scrutiny of the quality of the new activity (as gauged by curriculum, delivery, quality of faculty, and 
so on) following the requirements set out in the UPR Framework. 

Recommendation 6: The provisions governing the implementation of 
new programs at the University of Ottawa, as at all Ontario 
universities, should be revised consistently with both the Ministry of 
Education and Training information requirements that need not be 
supplied in detail but must be certified by the president of the 
university in funding applications as set out in the MET memorandum 
regarding the Program Approvals Process on November 8, 1996, and 
the UPR Framework found in Appendix A 1.4. 



7 Conclusion 

The Auditors wish to note that they believe the Undergraduate Program Review Audit Procedure 
reinforces and supports the commitment of Ontario’s universities to offering programs of the highest 
quality. It builds on the internal quality assurance processes that the institutions have put in place. 
Based on their own experience as senior academic administrators, the Auditors recognize the 
importance of such processes in maintaining and improving the quality of education that universities 
deliver to undergraduates. Also, they appreciate the difficulty the institutions face in times of fiscal 
constraint. The pressures are immense. 

It is to the credit of the institutions that they are committed to self-examination and renewal in order 
to offer programs of excellent quality. A regular, institution-wide series of undergraduate program 
reviews is a serious and demanding undertaking. These reviews are not the only ones that 
departments and other academic units undergo, and the demands on the faculty and staff are 
considerable. For this reason, the Auditors have been most impressed by how thoroughly and 
carefully Brock University and the University of Ottawa, the two institutions that were audited in 
1997, have approached their undergraduate review process. 
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As a general conclusion, the Auditors are satisfied that, in regard to both policy and practice, the 
University of Ottawa’s process for undergraduate program reviews conforms closely to the UPR 
Framework. Indeed, the Ottawa review policy is, in their view, exemplary and, as already observed, 
the university’s practice goes beyond what is required by its policy. They have noted with 
recommendations (mostly having to do with the role of the Senate committee, SCEUP, and follow-up 
to the reviews) and suggestions where the policy might be improved and made to conform even more 
closely with the UPR Framework. 



8 Summary of Recommendations and Suggestions 

Finally, for ease of reference the Recommendations and Suggestions are repeated here: 

Recommendations: 

Recommendation 1: Though there is no evidence to question the impartiality of the external 
evaluators who have been chosen by the University of Ottawa, the auditors recommend that all 
Ontario universities that have not already done so enunciate as part of their program review 
policy the principle that external evaluators will be at arm’s length from the program under 
review, along with their interpretation of what is entailed by the principle (not relatives, 
collaborators, supervisors/supervisees, and so on). 

Recommendation 2: If the Senate Committee on the Evaluation of Undergraduate Programs is 
to play the role of an appraisal committee of peers from within the university, its terms of 
reference must be explicit about that role and function. In fulfilling that role, the Senate 
Committee must review the self-study brief, the external evaluators’ reports as well as the 
department’s response in order to arrive at a conclusion on the quality of each program and 
determine the actions to be taken as a result of the review. 

Recommendation 3: The Ottawa evaluation policy should be revised to make explicit provision 
for “procedures for action on the recommendations arising from the undergraduate program 
review,” as required by the UPR Framework, and reflecting actual practice. 

Recommendation 4: The university’s policy should be revised to make clear that a summary 
report on the results of the internal review process will be presented annually to the Senate and 
the Board of Governors. In addition, the Senate Committee should make every effort to ensure 
that its reports deal with both the strengths and weaknesses of the programs that have been 
reviewed. 
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Recommendation 5: The university review policy should include a reference to the evaluation 
of the appropriateness of admission requirements where academically warranted. 

Recommendation 6: The provisions governing the implementation of new programs at the 
University of Ottawa, as at all Ontario universities, should be revised consistently with both the 
Ministry of Education and Training information requirements that need not be supplied in detail 
but must be certified by the president of the university in funding applications as set out in the 
MET memorandum regarding the Program Approvals Process on November 8, 1996, and the 
UPR Framework found in Appendix A 1 .4. 



Suggestions: 

Suggestion 1 : It is suggested that the Senate Committee on the Evaluation of Undergraduate 
Programs (SCEUP) and the Vice-Rector, Academic, together review the appropriateness of the 
requirement that internal reviews and OCGS appraisals and external accreditation exercises “take 
place during the same year,” if the purpose is to “make best use of the time, energy, and resources 
already required” for the two kinds of external process. 

Suggestion 2: Since all academic units are subject to internal reviews and since the provision and 
the analysis of data is an important component of the quality evaluation, it is suggested that SERP 
develop a method for providing the required data in a simplified and at least partially analyzed 
format, based on common algorithms for all units. SIRP might also make workshops available 
on reading and understanding the material it produces, and the kinds of questions that can be 
asked of the data. 

Suggestion 3: Since the practice at Ottawa has been to retain external evaluators in all cases, it 
is suggested that the word “normally” be removed from the policy document. This would 
strengthen the policy and remove any possible doubt as to the intentions of the university. 

Suggestion 4: In a context where all programs within the university are to be assessed within a 
seven-year cycle, it would make sense for programs under review and external evaluators to be 
enjoined to consider critically whether or not the needs of students within the program being 
reviewed are being met by the courses either required or recommended in other departments of 
the university. 

Suggestion 5: In reviewing its policy, the university should consider making available to 
programs undergoing review the option of seeking advice from representatives of industry, 
professions and practical training programs when it is regarded as useful by the unit under review. 
If this option is introduced, the policy should make it clear that there must always be at least one 
evaluator who is an academic even when the advice of others is sought. 
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Appendix 1 — Schedule for Program of Audits in First Cycle: 1997-2003 



Year in 


:? Calendar 




^Universities 




1 


1997 


Brock 


Ottawa 


— 


2 


1998 


Western 


Windsor 


— 


3 


1999 


Carleton 


Queen’s 


Nipissing 


4 


2000 


Lakehead 


York 


— 


5 


2001 


Ryerson 


Toronto 


Trent 


6 


2002 


Laurentian 


Waterloo 


— 


7 


2003 


Guelph 


McMaster 


Wilfrid Laurier 
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Appendix 2 — List of the Names of the Auditors 
Dr. Naomi Griffiths: 

Adjunct Research Professor, Department of History, Carleton University; Dean, Faculty of 
Arts, Carleton University, 1979-87 

Dr. Ross Rudolph: 

Professor, Department of Political Science, York University; Associate Dean, Faculty of Arts, 
1985-88; Associate Vice-President (Faculties), York University, 1990-1996 

Dr. Edward Stansbury: 

Professor Emeritus, Meteorology, McGill University; Dean, Faculty of Arts and Science, 
1969-71; Dean, Faculty of Science, 1971-76; Vice-Principal, Planning, McGill University, 
1976-86; Member, la Commission de verification de 1’evaluation des programmes, Quebec 
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Appendix 3 — List of the Principal Documents Provided by the University of 
Ottawa 



Evaluation of Undergraduate Programs. Policy document approved by the Senate. November 2, 
1992. 

Evaluation des programmes de ler cycle. Schedule for undergraduate program reviews, 1993-2000. 
le 20 novembre 1996. 

Evaluation des programmes de ler cycle. Rapport pour la periode 1995-1996 du Comite du Senat 
sur 1’evaluation des programmes de ler cycle, le 30 septembre 1996. 

Council on Undergraduate Studies. Terms of Reference. January 22, 1990. 

Documentation for the reviews of the undergraduate programs in Chemistry (1993-94), English 
(1995-96) and Physiotherapy (1993-94) includes: departmental self-evaluation of the program, 
reports from external evaluators, letters containing comments of the chair of the department and 
dean of the faculty, the report of the Senate Committee on the Evaluation of Undergraduate 
Programs, and the follow-up letter from the Vice-Rector, Academic, to the chair of the 
department. 

Documentation for the implementation of the new undergraduate program in Environmental Studies 
includes: “Proposal for a New BA Honours Program in Environmental Studies (Bilingual) at the 
University of Ottawa”, and extracts from minutes of the meetings of Council on Undergraduate 
Studies, the Executive Committee of the Senate and the Senate. 1994-95. 
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Appendix 4 - 


— Schedule for the Site Visit 


8:00 a.m. 


Dr. Bernard Philogene, Vice-Rector, Academic and Research, and 
Dr. Grilles Patry, Vice-Rector, Academic-designate. 


9:00 a.m. 


Dean Monique Begin, Faculty of Health Sciences 
Dean Hugh French, Faculty of Science 
Dean David Staines, Faculty of Arts 


10:00 a.m. 


Representatives of the Physiotherapy Program: 

Professor Joan McComas, Program Director 

Ms. Dawn Burnett, Assistant to the Director (at the time of the evaluation) 


11:00 a.m. 


Representatives of the Department of English: 

Professor Camille La Bossiere, Chair (at the time of the evaluation) 

Professor Bernhard Radloff, Director of Undergraduate Studies (at the time of the 
evaluation) 

Ms. Karen Selesky, a student who met with the appraisers at the time of the visit 
Professor Nicholas von Maltzahn, Director of Undergraduate Studies 


2:00 p.m. 


Representatives of the Department of Chemistry: 

Professor Christian Detelier, Chair 

Professor Barry A. Morrow, Acting Chair (at the time of the evaluation) 
Professor Tony Durst, Department of Chemistry 


3:00 p.m. 


Meeting with the members of the Senate Committee on the Evaluation of 
Undergraduate Programs: 

Professor Alain Bisson, Faculty of Law 

Professor Jacqueline Bosse-Andrieu, Faculty of Arts 

Professor Denis Carrier, Assistant Vice-Rector, Academic, Resource Person 

Professor Professor Fran<?ois Chapleau, Faculty of Science 

Professor Serge Denis, Faculty of Social Sciences 

Mrs. Lise Lepine, Assistant to the Vice-Rector, Academic, Secretary to the Committee 

Professor Margaret McKinnon, Faulty of Education 

Professor Bernard Philogene, Vice-Rector, Academic and Research, Chair 
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